I. Field of the Invention
This disclosure relates generally to apparatus and methods for augmented reality and other computer vision application, and more particularly to integration of camera auto-focus with computer vision-based recognition and tracking.
II. Background
Augmented reality systems use natural features as reference points within a sequence of images to place computer generated icons and images. A natural feature processing engine, including a natural feature detection module and a natural feature tracking module, is used to find and follow these reference points. Mobile devices may be enhanced with such augmented reality engines. Many mobile devices also have cameras with auto-focus capabilities provided by an auto-focus engine. Both natural feature and auto-focus engines track changes from image to image, however, known systems fail to allow communication between these engines.
In augmented reality, tracking that accurately follows the tracked object's movement and position creates a significantly improved user experience. Consequently, much effort is put into improving tracking performance. Object tracking functionally in a processor operates separately from auto-focus functionality at the front end of a camera. Auto-focus functionality is typically performed in hardware or with hardware acceleration. Auto-focus operations may result in information useful for improving natural feature detection and/or tracking. Similarly, natural feature detection and tracking may result in information useful for improving auto-focus functionality.
Many existing mobile devices 10 contain a camera and a processor. The camera provides images to the processor, which may modify the image by various augmented reality techniques. The processor may send a control signal trigger to camera activation and the camera provides the image or sequence of images to the processor for image processing in response. No information obtained from natural feature processing is returned to the camera to assist in obtaining an improved image. That is, control information beyond triggering does not flow from the processor to the camera.
In other existing mobile devices 10, image processing associated with natural feature detection and tracking is disassociated with image processing associated with auto-focusing. FIG. 1 shows a known system containing a natural feature processing engine 110 and an auto-focus engine 300, which are uncoupled and therefore do not communicate information as shown by delineation 400. An existing mobile device 10 contains one or more processors that function as a natural feature processing engine 110 and also as an auto-focus engine 300. The natural feature processing engine 110 includes a natural feature detection module 120 and a natural feature tracking module 125.
In general, operations in the natural feature detection module 120 and the natural feature tracking module 125 function in parallel, however, for a particular natural feature, these operations appear to occur in sequence where a natural feature is first detected within an image then tracked through subsequent images. The location of the natural feature within the image is used by a separate processing for augmented reality module 130. Each image undergoes processing through the natural feature detection module 120 to detect new natural features and also undergoes processing through the natural feature tracking module 125 to follow the movement of already detected natural features from image to image.
As shown at delineation 400, the auto-focus engine 300 has no communication with the natural feature processing engine 110 and may run as a parallel task. The auto-focus engine 300 may be implemented in hardware or may be implemented in a combination of hardware and software. The auto-focus engine 300 operates in real-time or near real-time to capture new images. Thus, a continued need exists to improve both natural feature processing as well as auto focusing.